Fast Sketch Recognition Using Stroke Constraint Indexing

Sketching is a natural modality of human-computer interaction for a variety of tasks, including, conceptual design. Our goal is to make drawing on a computer, as close as possible to drawing on a piece of paper, by placing as few constraints on the user as possible, while still enabling computer recognition of the hand-drawn shapes. Systems should include the ability to draw shapes in any order, and to allow interspersing (i.e., starting one shape, stopping, drawing another shape, then finishing drawing the initial shape). However, allowing interspersing complicates sketch interpretation. If each shape were completed before the next was started, segmentation and interpretation need to consider only a set of chronologically contiguous strokes. In the presence of interspersing, every subset of shapes on the screen has to be examined. Many sketch systems deal with this by disallowing interspersing beyond a limited amount of time or number of strokes; those that don’t may experience exponential slowdown when many possible subshape candidates are on the screen. We believe that placing such constraints on the sketcher is unnatural. We present here an indexing technique that takes advantage of the LADDER constraint language to index each shape for efficient access later. While our recognition algorithm is still exponential, this indexing technique removes all computation except list retrieval and comparisons to improve average-time performance and to support near real-time recognition even when all possible shape subsets are used as shape candidates. This removes constraints on drawing order, providing for more natural sketch interaction. We present empirical timing data results and showing that with our technique, even with over a hundred subshape candidate lines on the page, the system recognizes new shapes in less than a second. ACMClassification: H5.2 [Information interfaces and presentation]: User Interfaces Graphical user interfaces. General terms: Design, Human Factors, Algorithms, Languages.